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1.  SUMMARY 

This  paper  will  present  a methodology  for  computerised 
evaluation  of  camouflage  effectiveness.  The  methodology  is 
implemented  in  software  at  Danish  Defence  Research 
Establishment  (DDRE)  under  the  acronym  CAMEVA.  Basic 
input  is  a single  image  comprising  a highly  resolved  static 
target  as  well  as  a proper  amount  of  representative 
background.  Separate  target  and  background  images  can  also 
be  handled.  Target  and  background  regions  are  manually 
selected  using  the  computer's  standard  pointing  device  (i.e.  the 
mouse).  From  the  input  data,  CAMEVA  predicts  the  target 
detectability  as  a function  of  the  target  distance.  The 
detectability  estimate  is  based  on  statistical  distributions  of 
features  extracted  from  the  imagery,  establishing  a 
multidimensional  feature  space.  In  the  feature  space,  the 
Bhattacharyaa  distance  measure  is  applied  as  an  estimator  of 
the  separability  between  the  target  and  the  background.  The 
intention  is  that  the  extracted  features  should  resemble  those 
applied  during  the  human  perception  process.  Typically, 
contrast  and  various  measures  of  edge  strength  are  applied. 

The  Bhattacharyaa  distance  establishes  a relative  separability, 
while  the  absolute  detection  range  is  obtained  by  deriving  a 
relation  between  the  Bhattacharyaa  distance  and  the  estimated 
target  resolution,  at  range.  Thus  by  introducing  parameters  of 
the  sensor,  typically  the  human  unaided  eye,  detectability  as  a 
function  of  the  range  is  obtained.  The  methodology  will  not 
reflect  individual  observer  performance  but  is  aimed  at 
providing  an  estimate  of  the  optimal  detection  performance, 
given  the  selected  set  of  features.  During  the  choice  of  features 
and  of  sensor  parameters,  other  perception  mechanisms,  than 
the  human  observer  performance,  can  be  modelled  with  this 
methodology.  The  paper  will  discuss  theoretical  and  practical 
aspects  of  CAMEVA.  Validation  and  application  examples, 
including  results  on  the  NATO  RTO/SC1-012  SEARCH_1  and 
SEARCH_2  datasets,  will  be  presented  together  with  other 
data. 


Keywords:  evaluating  camouflage  effectiveness,  target 
detectability,  detection  range  estimation,  Bhattacharyaa 
distance,  SEARCH_1  and  SEARCH_2  data  analysis 

2.  INTRODUCTION 

CAMEVA  is  a methodology  developed  at  the  Danish  Defence 
Research  Establishment  (DDRE)  for  computerised 
CAMouflage  EVAluation  and  for  estimation  of  target 
detectability.  Input  is  a single  digitised  image  comprising  a 
highly  resolved  target  as  well  as  a proper  amount  of 
background.  Based  on  that,  CAMEVA  predicts  the  target 
detectability  as  a function  of  range,  in  principle  from  relatively 
close  range  to  infinite. 


The  methodology  is  based  on  measuring  the  dissimilarity 
between  the  statistical  distributions  of  features  on  the  target 
and  on  the  background. 

The  need  for  objective  and  cheap  methods  for  the  evaluation 
of  the  effectiveness  of  camouflage  measures  was  the  original 
motivation  for  the  development  of  CAMEVA.  The 
methodology  has  however  been  applied  in  other  tasks  as  well, 
including  characterisation  of  sensors,  evaluation  of  image 
processing  algorithms  and  multiple  sensor  fusion  algorithms. 

2.1.  History 

In  the  late  seventies  and  early  eighties  it  was  recognised  at 
DDRE,  that  digital  image  processing  was  becoming  available 
as  a useful  tool  for  solving  many  tasks  of  relevance  to  the 
military.  Clearly  it  was  important  also  to  consider  image- 
processing  methods  in  relation  to  the  evaluation  of  camouflage 
effectiveness. 

The  advantages  were  obvious.  If  such  methods  were  available, 
the  camouflage  effectiveness  could  be  assessed  without  the 
presence  of  costly  equipment  and  significant  numbers  of 
human  observers,  as  it  is  needed  in  field  trials  and  in  photo- 
simulation experiments.  Evidently  there  were  attractive 
economical  aspects  of  this,  but  also  the  technical  and  scientific 
aspects  were  considered,  such  as  the  desire  for  speedy, 
reproducible  results,  that  are  easily  documented. 

For  various  reasons  camouflage  activities  at  DDRE  were 
reduced  to  a minimum  during  a period  of  the  late  eighties  and 
early  nineties.  The  existing  software  was  “mothballed”  and 
further  development  within  the  regime  of  DDRE  was 
cancelled  during  that  period. 

New  tasks  related  to  camouflage  evaluation  and  to  the 
estimation  of  detectability  have  however  led  to  a resumed 
activity  within  this  field.  The  old  work  was  brushed  up  and 
that,  together  with  new  research  in  the  field,  led  to  the 
development  and  implementation  of  CAMEVA. 

3.  SYSTEM  OVERVIEW 

The  aim  with  CAMEVA  is  to  model  the  potentially  achievable 
performance  of  a detection  task.  The  aim  is  not  to  provide 
detailed  models  of  the  functionality  of  the  very  complex 
physiological  and  intellectual  processes  related  to  a human 
detection  task.  The  aim  is  rather  to  establish  limits  to  the 
performance  that  can  be  achieved,  based  alone  on  the 
information  hidden  in  the  scenario.  Thus  CAMEVA 
establishes  an  upper  limit  for  the  probability  of  detection. 

A detection  task  is  most  often  carried  out  by  observers  using 
the  unaided  eye,  but  this  particular  methodology  is  also 
applicable  to  observers  using  electro-optical  equipment  and  as 
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a reference  metric  for  optimality  daring  the  design  of  detection 
algorithms. 

The  high-level  diagram  of  Figure  1 shows  the  main  building 
blocks  of  CAMEVA. 


Im^ge 


Figure  1:  The  main  building  blocks  of  CAMEVA. 

The  input  is  an  image  presumed  to  be  of  good  quality,  l.e.  the 
image  should  have  a high  resolution  compared  with  the 
degradations  caused  by  the  sensory  system  being  modelled, 
and  atmospheric  transmission  effects  should  be  negligible. 

This  assumption  is  needed  due  to  the  simulation  strategy 
applied  in  CAMEVA.  The  results  should  be  limited  by  the 
system  parameters,  and  not  by  the  quality  of  the  input  data. 

The  first  processing  block  is  the  feature  extraction.  The  target 
strength  is  based  on  the  measurement  of  the  strength  of  certain 
features  on  the  target  relative  to  the  strength  of  these  features 
on  the  background. 

The  choice  of  which  image  regions  belong  to  the  target  and 
which  regions  belong  to  the  background  is  based  on  an 
interactive  selection  procedure  carried  out  by  the  system 
operator. 

The  choice  of  features  depends  on  the  characteristics  of  the 
presumed  perception  mechanism.  If  the  unaided  human  eye  is 
applied,  features  corresponding  to  those  applied  during  the 
human  perception  process  should  be  used.  Or  if  some  other 
perception  mechanism  is  assumed,  the  features  corresponding 
to  that  mechanism  should  be  applied.  A typical  choice  of 
features  in  the  case  of  the  unaided  human  eye  is  contrast, 
texture,  shape,  and  edge-content  sensitive  features. 

This  is  not  a perception  model  in  the  more  rigorous  definition 
of  this  term,  rather  than  it  is  a type  of  preprocessing 
attempting  to  extract  the  same  attributes  from  the  data,  as 
those  used  during  the  processing  performed  by  the  actual 
perception  mechanism.  In  the  current  context  we  will  however 
denote  this  processing  "the  perception  model"  since  this  is  the 
step  in  the  processing  where  the  characteristics  of  the  psycho- 
visual processes  arc  being  modelled. 

Based  on  the  features,  the  analysis  kernel  estimates  the 
statistical  distribution  of  the  features  of  the  target  and  of  the 
background.  A measure  of  the  difference  between  the 
distributions  is  established.  This  provides  a relative  measure  of 
detectability,  i.e.  a measure  that  is  independent  of  the  target 
range.  Finally  the  relative  difference  measure  is  weighted 
according  to  the  degradation  introduced  by  the  limited 
resolution  of  the  sensor  and  thereby  also  introduces  the  target 
distance  as  a system  parameter. 


Atmospheric  influences  on  the  results  arc  currently  not 
implemented.  Consequently  the  underlying  assumption  is  ideal 
atmospheric  conditions.  This  is  recognised  as  an  important 
issue  for  the  continuing  improvement  of  the  system.  The  idea 
is  to  propose  an  atmospheric  weight  factor  to  the  measure  of 
separability;  similar  to  the  way  the  sensor  characteristics  are 
modelled  into  the  system.  It  has  been  demonstrated2  that  this, 
under  certain  conditions,  can  be  done  with  a simplified 
transmission  loss  model. 

The  perception  model  typically  used,  is  a simplified  model  of 
human  vision,  based  on  contrast  and  edge  detection.  The 
method  applied  in  the  analysis-kernel  is  based  a statistical 
approach  for  measuring  the  separability  between  distributions, 
known  as  the  Bhattacharyaa3  distance. 

The  following  sections  will  discuss  various  aspects  of  this 
methodology.  The  statistical  decision  theory  involved  is 
discussed  initially,  since  this  is  also  applied  to  illustrate  the 
motivation  for  the  choice  of  features.  This  discussion  is 
followed  up  later,  w ith  considerations  of  atmospheric 
transmission  and  light  conditions.  Validation  examples  are 
shown  at  the  end  of  the  paper. 

4.  STATISTICAL  DECISION 

The  statistical  decision  procedure  is  based  on  a feature  vector 

x = {a,,  x: x„).  Each  x represents  one  feature.  We  claim 

that  detection  is  possible  when  the  multidimensional 
distribution  function  of  features  on  the  target  is  sufficiently 
different  from  the  corresponding  background  distribution. 


I 


Figure  2:  Distribution  densities  of  target  and  background, 
with  decision  errors  represented  by  r„  F/,  and  the  decision 
threshold  r„. 

We  further  claim  that  estimates  of  the  target  and  background 
distributions,  based  on  high-resolution  imagery,  can  be  utilised 
to  predict  the  corresponding  distributions  as  a function  of  the 
distance.  That  is  the  same  as  postulating  that  basically  the 
distributions  arc  inherent  properties  of  the  target  and  the 
background.  For  the  moment  we  shall  neglect  any  external 
influence  (i.e.  atmosphere  etc.)  that  violates  this  assumption. 

4.1.  Basic  decision  theory 

The  target  and  background  distributions  are  exemplified  for  a 
single  feature  in  Figure  2.  In  the  detection  case,  assuming  a 
onc-samplc  observation,  the  fraction  of  the  distribution 
overlap  denoted  F,  represents  the  error  of  missing  a target  with 
r„  as  the  detection  threshold.  Hence  the  probability  of  missing 
the  target  is  P(b)r{,  with  P(b)  as  the  a priori  probability  of  a 
random  sample  being  background.  The  errors  represented  by 
Fh  (i.e.  the  false  alarm  rate)  are  irrelevant  in  this  situation 
where  we  arc  evaluating  a known  target  relative  to  the 
background.  Correspondingly  the  probability  of  detection  P(d) 
is: 
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P{d)  = \-P(b)T,  (1) 

In  general,  with  an  ^-dimensional  distribution  Ft  is  determined 
over  the  n dimensions  of  the  observation  vector. 

By  introducing  the  sampling  and  assuming  k independent 
samples  of  the  target,  the  detection  probability  becomes: 

P{d)  = \-P{b)Yf  (2) 

4.2.  The  Bhattacharyaa  Distance 

Given  that  parametric  distributions  are  unknown,  together 
with  the  general  problems  of  analytically  determining  the 
distribution  overlap  r=rl+rh,  we  introduce  the 
Bhattacharyaa3  distance  D,  that  is  a measure  of  how  different 
two  distributions  are: 

y.r  (x)//,  (x Vx  i (3) 

-M  J 

The  integral  is  an  approximation  to  T. 

In  the  detection  case  where  the  target  qua  the  distance 
occupies  only  a small  fraction  of  the  total  field  of  view  (FOV), 
a reasonable  approximation  to  P{b)  is  P(b)&  1.  Furthermore  it 
follows  from  P(6)«  1 that  rb  is  negligible  compared  to  Tt  and 
thus  rt  is  an  approximation  to  the  total  distribution  overlap. 

Using  P{b)*\  and  Fti=r,  a minor  rewrite  of  equation  (2) 
provides: 

P(d ) « 1 - exp  {k  In  {r}}  (4) 

The  distribution  overlap  is  represented  by  the  integral  within 
the  Bhattacharyaa  distance,  thus  Lte-ln{F).  In  the  limiting 
case  of  identical  distributions  it  is  easily  seen  thatD=0,  and  of 
totally  different  distributions  that  D ->oo. 

With  the  Bhattacharyaa  distance  we  have: 

P(d)*l-exp{-kD}  (5) 

We  see  that  in  the  case  of  identical  distributions  P(d)~ 0,  which 
is  reasonable  due  to  P(b)= 1.  This  means  that  there  is  virtually 
no  chance  of  hitting  the  target  by  random  choice.  Similarly  by 
totally  different  distributions  P(d)= 1,  which  is  also  reasonable 
since  this  allows  detection  of  an  almost  infinitely  small  target. 


4.3.  Limited  Resolution 

We  introduced  k as  the  available  number  of  independent 
samples  of  the  distribution.  In  the  visual  detection  task  k will 
depend  on  the  target  size  A,  the  target  distance  L and  how  well 
the  eye  is  capable  of  independently  sampling  the  FOV.  The 
last  parameter  we  shall  denote  "the  effective  minimum 
resolvable  field  of  view"  9k.  The  target  size  is  normally  trivial 
and  the  target  distance  is  a variable  parameter  of  the 
simulation  (i.e.  we  are  aiming  at  detection  curves  as  a function 
of  the  distance).  Thus  with  the  assumption  that  9k  is  also 
known,  the  target  resolution  k is  determined  by  the  geometry 
of  the  scenario  and  we  obtain  the  probability  of  detection  as  a 
function  of  the  range: 


Where  k is: 

KL)  = (7) 

In  the  case  of  different  horisontal  and  vertical  resolution 
characteristics  6k2  is  replaced  by  the  product  of  the  separated 
horisontal  and  vertical  components  9^  and  6k  v. 

4.4.  Multidimensional  Feature  Spaces 

Numerically  computation  of  the  Bhattacharyaa  distance  in  the 
multidimensional  feature  space  is  quite  complex  and  in  fact 
not  necessary.  From  the  definition  of  the  Bhattacharyaa 
distance  we  obtain  with  the  assumption  of  independent 
features  that: 

£>  = SA,  on 

/= l 

We  may  therefore  conveniently  write  D as  the  sum  of  the 
individual  Bhattacharyaa  distances  for  each  feature. 

This  result  is  important  since  it  implies  that  the  complexity 
involved  with  the  computation  of  estimates  of 
multidimensional  distributions  can  be  avoided. 

Whether  this  assumption  is  valid  depends  on  the  choice  of 
features. 

4.5.  Feature  extraction 

Selection  of  features  is  a nontrivial  task.  This  is  illustrated  in 
Figure  3.  On  the  upper  left  "triangles"  image,  the  two  “targets” 
are  easily  detected  visually.  A statistically  based  metric, 
operating  solely  on  the  contrast,  provides  zero  difference 
between  both  two  targets  and  the  background. 

Clearly  other  features  are  also  involved  with  the  detection 
process  in  the  example,  although  the  contrast  is  still  relevant. 
The  three  additional  features  in  the  Figure  illustrates  this: 

It  is  evident  that  each  of  the  features  reflects  different 
properties  of  the  input  data.  We  will  consider  the  target  and 
background  regions  of  the  added  features: 

The  strength  of  the  horizontal  edges  are  applicable  for 
detection  of  the  upper  right  target  (which  we  will  denote 
“target  A”),  as  these  edges  are  relatively  strong  in  the 
background,  and  virtually  non-existent  in  the  target.  The  lower 
left  target  (denoted  “target  B”)  is  more  difficult  to  detect,  and 
the  only  useful  cue  is  missing  edge  at  the  top  of  the  target. 

With  respect  to  the  vertical  edges,  target  A is  once  again  easily 
detected.  This  time  due  to  the  strong  vertical  edges. 

Target  B is  difficult,  although  still  detectable,  due  to  the 
discontinuities  of  the  edges.  But  basically  the  edge-strengths 
of  both  the  target  and  the  background  are  the  same. 

With  respect  to  the  isotrope  edge  features  both  targets  are  - 
roughly  speaking  - equally  difficult  to  detect. 

We  see  that  the  choice  of  features  is  vital  to  the  result.  In  the 
case  of  an  analysis  (i.e.  an  algorithm)  applying  only  contrast 
processing,  this  single  feature  might  be  sufficient.  But  in  other 
situations  more  complex  features  should  be  applied. 


DA 

l2oi ; 


P(d,L)  = 1-exp 


(6) 
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Figure  3:  Four  different  features  of  a simple  image 
with  two  target.  Both  the  targets  and  the 
background  arc  triangles.  The  unprocessed  (the 
contrast  feature)  and  three  additional  features  based 
on  edge  strength. 

By  application  of  the  relative  analysis-part  ofC'AMFVA.  (i.e. 
computing  the  Bhattacharyaa  distance),  to  the  triangle  input 
image  with  all  four  features,  the  computational  results  should 
reflect  the  qualitative  assessment  above.  The  results  arc  shown 
in  table  I . 

The  Bhattacharyaa  distances  correspond  nicely  to  the  results 
obtained  from  the  visual  assessment:  D( ■ is  about  the  same  for 
both  targets,  which  means  that  based  on  contrast  both  targets 
are  equally  easily  detected.  and  Dy each  are  relatively 
strong  in  target  A.  while  is  weak  in  B.  The  isotrope 
component  of  the  target  is  about  the  same  for  both  targets, 
which  is  also  reasonable  since  the  have  equal  components  of 
diagonal  edges. 

Typical  features  that  have  been  applied  in  the  application  of 
CAMF.VA  to  visual  detection  tasks  arc  the  four  features  used 
in  this  example. 

4.6.  Atmospheric  and  light  conditions 

While  /((/,)  provides  a method  of  limiting  the  resolution  due  to 
distance,  it  does  not  model  the  effect  of  atmospheric 
transmission  loss  and  light  conditions  (i.e.  day  or  night).  The 
current  work  will  in  principle  assume  infinite  visibility  and 
full  daylight.  A simple  model  for  the  atmospheric  influence  on 
the  Bhattacharyaa  distance  have  been  considered",  providing  a 
Bhattacharyaa  distance  as  a function  of  /,.  Likewise  a simple 
model  for  the  influence  of  the  light  is  considered.  Non  of  these 
modelling  concepts  have  however  been  validated. 

4.7.  Training  of  algorithm 

Training  of  CAMF.VA  is  in  principle  a very  simple  procedure. 
There  is  only  one  parameter  (9k)  that  is  not  easily  determined, 
neither  by  the  physical  setup  of  the  experiment,  nor  by  the 
optics  of  the  sensor. 

The  parameter  Gk  depends  on  psychological  factors  such  as 
motivation  and  on  experience  with  visual  target  acquisition. 


Table  1:  Target/background  Bhattacharyaa  distances  for 
the  four  different  features  of  the  triangle  image. 


Feature 

Target 

Contrast 

|/VI 

Horizontal 
edge  |0„| 

Vertical 
edge  \Dy\ 

Isotropc 
edge  [I),] 

A 

0.0336 

0. 1 1 20 

0.0894 

0.0666 

B 

0.0363 

0.0140 

0.0501 

0.0556 

Based  on  the  physiology  of  the  eye1,  figures  in  the  order  of  0.2 
mr  are  obtained.  In  the  detection  task  however,  where  the 
observer  is  unable  to  focus  on  the  target,  this  figure  is 
typically  too  optimistic.  Fxpcrimental  data1  prepared  at  DDRF 
have  shown  that  #*=2.0  mr  is  a useful  value  for  a group  of 
non-military  observers.  In  the  case  of  highly  trained  observers, 
other  values  of  8k  may  have  to  be  used. 

4.8.  Practical  aspects 

To  illustrate  that  the  methodology  applied  in  CAMFVA  is 
quite  widely  applicable,  we  will  discuss  a few  aspects  related 
to  the  practical  use  of  CAMFVA.  This  illustrates  however  also 
that  this  technique  requires  a skilled  operator,  with  a general 
understanding  of  the  problems  related  to  the  cffectivity  of 
camouflage  measures. 

• Data  collected  with  detectability  as  the  main  purpose  arc 
typically  long-range  images.  Often  data  taken  at  close 
range  and  with  the  target  in  the  actual  background  have 
not  been  collected.  In  those  cases  experiments6  with  the 
application  of  CAMFVA  have  been  made  with  separated 
target  and  background  images.  Clearly  there  arc  problems 
related  to  that  procedure,  but  attempts  have  been  made  to 
align  the  internal  target  image-dynamics  as  well  as  the 
target-background  image-dynamics  in  this  type  of  data, 
thus  allowing  the  target  and  background  statistics  to  be 
taken  from  different  images.  Preferably  however  this 
procedure  should  be  avoided,  and  should  also  be 
avoidable  in  experiments  designed  specifically  as  input  to 
CAMFVA. 

• CAMF.VA  is  a human-in-the-loop  process.  With  the 
input  imager}',  the  operator  must  decide  which  image 
regions  to  apply  as  target  and  which  to  apply  as  the 
background.  An  example  of  this  procedure  is  illustrated  in 
Figure  4.  Clearly  that  procedure  is  sensitive  to  the 
selection  of  in  particular  the  background  region,  which  is 
not  well  defined,  and  the  operator  must  be  careful,  in 
order  to  produce  meaningful  results.  Related  to  that  is 


dataset,  with  overlay  of  target  and  background  regions 
applied  by  CAMEVA. 
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also  the  fact  that  the  results  are  basically  a kind  of 
averaging  across  the  target  region  and  the  background 
region.  I.e.  if  the  target  contains  isolated  bright  spots, 
acting  as  cues  to  the  observer,  the  averaging  is 
problematic. 

• In  a classical  camouflage-scenario  the  most  efficient 
camouflage  blends  the  target  into  the  local  background. 
CAMEVA  is  basically  designed  with  that  scenario  in 
mind.  In  other  scenarios,  for  example  desert  with 
scattered  trees  and  bushes,  the  most  efficient  camouflage 
will  sometimes  be  one  that  most  closely  matches  the 
target  to  the  scattering  structure.  Thus  analysis  made 
against  the  local  background  in  that  case  are  meaningless. 
In  those  situations  the  background  region  used  as  input  to 
CAMEVA  must  be  selected  from  the  relevant  part  of  the 
images,  and  not  from  the  local  background,  like  for 
example  regions  with  trees  and  bushes. 

Other  scenarios  can  be  thought  of  where  there  are  problems 
related  to  this  kind  of  methodology.  In  most  cases  however  a 
skilled  operator  will  be  capable  of  producing  useful  results. 

5.  RESULTS 

Results  based  on  three  different  datasets  are  presented. 
Initially  we  discuss  the  JPRIS  dataset  that  was  specifically 
collected  as  a means  for  testing  and  calibrating  CAMEVA. 
Secondly  we  discuss  the  application  of  CAMEVA  to  the 
SEARCHJ  and  SEARCH_2  datasets. 


Figure  6:  The  types  of  panels  applied  as  targets, 
not  shown  to  scale.  The  panel  areas  vary  from 
0.20  m2  to  0.90  m2,  while  the  signature  elements 
vary  from  0.025  m2  to  0.26  m2. 


5.1.  The  JPRIS  dataset 

The  jpris  dataset  is  based  on  an  observer  field  test1  conducted 
by  DDRE. 

The  experiment  was  designed  as  an  observation  trial  with 
observations  performed  by  the  unaided  eye. 

Targets  consisted  of  12  artificial  panels  of  6 different  types, 
either  rectangular  or  trapezoidal  as  illustrated  in  Figure  6. 

Each  panel  was  covered  by  a piece  of  green  texture  mat  and  a 
signature  was  attached  to  the  surface,  to  allow  identification 
when  observed. 

The  background  was  a line  of  vegetation  consisting  of  trees 
and  bushes.  All  of  the  panels  were  presented  against  that  same 
background. 

The  total  length  of  the  observation  path  was  approximately 
600  m with  seven  observation  posts  from  long  range  (665  m) 
to  close  range  (100  m)  arranged  along  the  path. 

A total  of  40  observers  recorded  the  point  of  detection  and  the 
point  of  identification,  starting  at  long  range  and  approaching 
the  target  panels  along  the  observation  path.  The  observers 
were  "semi-trained"  in  the  sense  that  they  were  scientific  and 
administrative  civilian  personnel  working  for  the  Danish 
military  and  some  had,  due  to  their  scientific  duties,  some 
experience  with  observation  tasks. 

To  support  the  computational  analysis  slides  were  taken  from 
the  observation  posts.  Each  target  panel  was  analysed  with 
CAMEVA  and  detection  curves  obtained.  Similarly  detection 
curves  were  extracted  from  the  field  trial  results.  Comparison 
of  the  results,  as  averages  across  the  observer  population,  is 
illustrated  in  Figure  7. 

It  is  seen  that  the  theoretical  results  to  a very  high  degree 
describes  the  results  of  the  field-trial.  From  these  results  it 
seems  indeed,  that  the  human  acquisition  process,  under 
certain  conditions,  can  be  described  as  a statistically  process 
derived  from  first  principles. 

The  parameter  #*=2.0  mr  is  however  determined  from  the 
same  population  as  were  used  as  observers  during  the 
experiment. 

One  important  error  source  is  not  taken  into  account  during 
this  experiment,  namely  the  presence  of  eye-catching  cues  in 
the  background  close  to  the  target.  It  is  believed  that  the 
unexpected  result  for  target-panel  two  is  due  to  an  abnormal 
intensity  distribution  (very  dark)  in  the  immediate 
neighbourhood  of  the  target.  There  is  evidence  within  the  data 
that  indicates  that  the  observers  have  detected  the  background, 
and  not  the  target  itself. 

5.2.  The  SEARCH  datasets 

The  SEARCH_1  and  SEARCH_2  datasets  comprise  a total  of 
44  images  of  six  military  vehicles  recorded  at  different  target 
distances  and  with  the  targets  presented  against  different 
backgrounds. 


Figure  5:  Example  on  the  deployment  of  target  panels  in  the  JPRIS  experiment. 
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Figure  7:  Detection  curves  obtained  as  a result  of  the  DDRE  JPRIS  experiment  compared  with  results  of  CAMEVA. 
Theoretical  results  are  the  solid  curves,  while  the  experimental  results  are  the  dotted  curves 


The  only  difference  between  SEARC1M  and  SFARCH_2  is 
the  resolution  of  the  digitised  imagery,  that  is  1536  x 1024 
pixels  in  the  case  of  SEARCI  M and  6144  x 4096  in  the  case 
of  SEARCH2. 

Data  are  available  in  digitised  form  as  full  colour  images  as 
well  as  grey-level  images.  Each  scenario  is  available  as  a pure 
background  image,  as  well  as  an  image  with  the  target  in  the 
background.  Furthermore  closc-up  images  of  each  target  arc 
provided  at  three  different  aspects.  Ground  truth  is  provided  as 
binary  target  masks  with  each  image.  Target  distances  vary 
from  about  800  m to  6 km. 

The  SEARCH  data  are  collected  and  distributed  by  TNO- 
HFRI  , and  have  kindly  been  provided  for  the  application  of 
the  data  to  computational  techniques  estimating  human 
observer  performance  in  detection  tasks. 

TNO-HFRI  has  also  conducted  photo-simulation  observer 
tests  on  the  data,  providing  a baseline  for  the  testing  of 
computational  methods  for  prediction  of  detectability  and 
search  time.  A second  photo-simulation  experiment7  on  a 
subset  of  the  SEARCH  ! data  was  prepared  by  DCTA1 2 


1 TNO  Human  Factors  Research  Institute,  Soesterberg, 
The  Netherlands 

2 Defence  Clothing  and  Textiles  Agency,  Colchester, 
The  United  Kingdom 


CAMEVA  was  applied  to  the  44  images  of  the  dataset  and 
probability  of  detection  was  computed  as  a function  of  the 
target  range.  P(d)  at  the  actual  target  range  was  computed  and 
compared  with  the  photo-simulation  results.  CAMEVA  was 
not  trained  on  the  SEARCH  datasets  prior  to  its  application  to 
them. 

Cross  comparisons  of  the  results  from  CAMEVA,  the  TNO 
experiment  and  the  DCTA  experiment  arc  sumarised  in  the 
bar-graphs  of  Figure  8. 

As  described  CAMEVA  normally  provides  detection  curved 
as  a function  of  the  distance.  In  this  case  and  to  aid 
comparison  of  results  P(d)  is  computed  at  the  actual  target 
distance  only. 

6.  CONCLUSIONS  AND  FURTHER  WORK 

Through  the  development  and  the  study  of  CAMEVA,  we 
have  established  a correlation  between  results  obtained  by 
human  observers  in  the  detection  task  and  the  results  provided 
by  CAMEVA.  It  is  also  evident  however  that  several  aspects 
of  CAMEVA  need  further  research  to  provide  a more  robust 
system.  The  most  important  areas  are  summarised  below: 

CAMEVA  depends  strongly  on  the  skills  of  the  operator 
during  the  selection  of  target  and  background  regions.  Rather 
than  consider  an  automated  procedure  for  that,  which  we 
believe  is  practically  impossible,  it  is  considered  to  produce  a 
kind  of  catalogue  that  will  set  up  typical  scenarios  together 
with  proposed  operator  methodologies  to  cope  with  these. 


Figure  8:  P(d)  estimated  by  CAMEVA  applied  to  SEARCH_1  (C-ST)  and  SEARCH2  (C-S2), 
compared  with  observer  experiment  results. 
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It  is  considered  important  to  implement  a proper  procedure  for 
modeling  of  atmospheric  transmission  loss  and  oflight 
conditions.  Fundamentals  for  these  sub-models  have  been 
investigated,  but  still  need  validation. 

The  current  choice  of  features  is  not  necessarily  optimal. 
Certain  aspects  of  detection  are  currently  not  modeled.  A 
typical  example  is  the  cueing  provided  by  long  straight  lines. 
Further  features  need  to  be  investigated  and  in  some  cases 
algorithms  for  their  implementation  must  be  developed. 
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